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This paper presents a report of research on the evaluation 
of systemic reform of education in science, mathematics, engineering, and 
technology (SMET) , focusing on the Statewide Systemic Initiatives (SSIs) that 
have been funded by the National Science Foundation (NSF) since 1991. The 
main purpose of the study was to develop recommendations for the future 
evaluations of SSIs. Additional goals were; to determine the most important 
elements of systemic initiatives that must be captured in the process of 
evaluation, to indicate why these elements are important, and to suggest ways 
or means of capturing them. The methodology involved interviews with 
evaluators of systems initiatives, principal investigators, and officials at 
NSF. There were 19 participants. The principal findings are that; (1) the NSF 
should be commended for supporting the systemic reform of SMET; (2) either 
the field of evaluation will encompass the evaluation of systemic reform as a 
subset or an entirely new field will emerge; (3) the need to evaluate massive 
SSIs has led to serious management issues at NSF; (4) systemic reform of 
education has yet to develop a clear and consistent language for alignment of 
values, policies, and practices while accommodating and coordinating local, 
state, and federal interests; and (5) the future evaluation of the statewide 
systemic reform of SMET must show that measurable achievement gains have been 
realized by the intended audience, especially historically disadvantaged 
students. The paper includes recommendations based on the analysis of data. 

An appendix presents interview questions for the study. (Author/SLD) 
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Abstract 

This article presents a report of research on the evaluation of systemic reform of education in 
science, mathematics, engineering, and technology (SMET), in particular, Statewide Systemic 
Initiatives (SSIs) that have been funded by the National Science Foundation since 1991. The main 
purpose of the study was to develop recommendations for the future evaluation of SSIs. Additional 
goals were: to determine the most important elements of systemic initiatives that must be captured 
in the process of evaluation, to indicate why these elements are important, and to suggest ways or 
means of capturing them. The methodology involved interviews with evaluators of systemic 
initiatives, principal investigators, and officials at NSF. There were nineteen participants. 

The principal findings are: that NSF should be commended for supporting the systemic reform 
of SMET; that either the field of evaluation will encompass the evaluation of systemic reform as a 
subset or an entirely new field will emerge; that the need to evaluate the massive SSIs has led to 
serious management issues at NSF; that systemic reform of education has yet to develop a clear and 
consistent language for alignment of values, policies, and practices and at the same time to 
accommodate and coordinate local, state, and federal interests; and that the future evaluation of the 
statewide systemic reform of SMET must show measurable achievement gains have been realized 
by the intended audience, especially historically disadvantaged students. The article includes 
recommendations based upon the analysis of data and closing remarks. 
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Lessons Learned in the Evaluation 
of Statewide Systemic Initiatives 

by 

Charles Bruckerhoff, Ph.D. 

Curriculum Research & Evaluation 

Introduction 

The main purpose of this study is to develop recommendations for the future evaluation of 
statewide systemic initiatives to reform education in science, mathematics, engineering, and 
technology (SMET). Since 1991 these have been funded by the National Science Foundation (NSF). 
The National Institute for Science Education (NISE), located at the University of 
Wisconsin-Madison, sponsored this research project through a fellowship to the researcher. Dr. 
Norman Webb, a senior scientist at NISE, served as the systemic evaluation team leader. Additional 
goals of the study are: to determine the most important elements of systemic initiatives that must be 
captured in the process of evaluation; to indicate why these elements are important; and to suggest 
ways or means of capturing them. 

Methodology 

The overall intent of this study is to report the perceptions of practitioners and officials who are 
familiar with the current process of evaluating systemic initiatives. The collection of data, chiefly 
through telephone interviews with evaluators of systemic initiatives for most of the states that have 
received SSI awards, principal investigators, and officials at NSF, was the research method used. To 
do this, the researcher developed eight questions that focused on various aspects of the evaluation 
of systemic reform (see appendix). The data collection process took place during November and 
December 1996.' The interview protocol involved obtaining an individual’s willingness to 
participate in the study, sending a copy of the questions to the participant in advance of the 
interview, scheduling a convenient time, and conducting the interview. The initial time allotment 
for telephone interviews was thirty minutes. However, most interviews lasted for ninety minutes or, 



'Also, on several occasions between January 1996 and June 1997, the researcher traveled to 
Madison, Wisconsin to conduct literature searches at the University of Wisconsin’s Memorial 
Library and to participate in lively discussion forums with other NISE Fellows, members of the 
Strategies for Evaluating Systemic Reform team, officials from NSF, and colleagues who work in 
the field of evaluation. 
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in several cases, for two or more hours. The participants prepared their answers in advance of the 
interview. During the interview, participants provided their prepared responses and also volunteered 
additional information. In one instance, an evaluation team organized a conference call, so that the 
researcher could obtain data through a small-group discussion. 

All the participants expressed a strong desire to help provide guidance for the future evaluation 
of systemic initiatives. Some participants said that the interview process was cathartic. It not only 
gave them an opportunity to describe and explain what happened in a particular state, but it also 
relieved them of self-doubts about their involvement in systemic evaluations. The latter effect was 
especially important to evaluators of states that were “defunded” by NSF. 

There were nineteen participants. Initially, the researcher recorded and transcribed the data in 
accordance with the questions of the interview protocol. Later, he analyzed the data for themes and 
supporting details. To a large extent, the results of this study are made up of the statements provided 
by the participants— with many of them expressing strong agreement on most issues. 2 

Systemic Reform 

“Never tell people how to do things. Tell them what to do and they will surprise you 
with their ingenuity. ” —George Smith Patton, 1885-1945 
What does systemic mean? The word systemic literally refers to that which is related to, or 
common to, a system or complex of systems, 3 such as the human body. The body’s systems may be 
whole or discrete parts, such as the nervous system or brain, blood circulation or heart, and so on. 
In this adjectival form, systemic denotes the flow or at least the passageway of something vital; 
hence, the importance of the nervous system simultaneously to human thought processes and 
physical movement. Conditions may exist in the organism that are either healthy or unhealthy. Also, 
external elements may be fed into one or more systems which promote a particular response that can 
affect one or more systems. For example, a systemic pesticide is a toxic substance that kills insects 
by interfering with their normal metabolic processes— systemically— and, thus, assures greater crop 



2 The paucity of information provided by the study participants on the second question under 
“Evaluation of Systemic Initiatives (see Appendix, p. 55) is due to the ambiguity and complexity of 
the question itself, participants said. Also, according to participants, the central issue regarding the 
evaluation of these systemic initiatives has been the manner and extent to which NSF has exercised 
its influence on the process. 

3 Systems theory, the foundation for systemic reform, was first proposed in the 1940's by the 
biologist Ludwig von Bertalanffy and was developed further by cyberneticist Ross Ashby. There 
have since been many new developments and applications of systems theory. For a study of the early 
work, see Ludwig von Bertalanffy, General System Theory: Foundations, Development, 
Applications (New York: G. Braziller, 1968); and William Ross Ashby, Introduction to Cybernetics 
(New York: J. Wiley, 1956). For recent applications, see these two works by George J. Klir Facets 
of Systems Science (New York: Plenum, 1991) and Architecture of Systems Problem Solving (New 
York: Plenum, 1985). 
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productivity or a lower incidence of pest-borne disease. Systemic antibiotic therapy is a form of 
treatment in oral surgery. 

Both positive and negative images are instructive. Because an organism (including all manner 
of social organisms) functions with systems, problems can be identified, understood, treated, and 
solved via those systems. By the same token, the growth or development of a huge social system, 
such as statewide education, can be enhanced by maximal use of its various systems. Thus, total 
systemic educational reform seeks to improve the education of children by introducing 
improvements throughout the parts of the system, such as changes in state legislation and policy, 
teachers’ professional knowledge and skill, content area standards, and standards-based assessment. 
Systemic reform implies the continuous renewal to adapt the educational system— that is basically 
good — to new and unforseen circumstances and, in the process, strive for significantly improved 
educational results. 

In these times, talk about reform of American society is common, especially in reference to 
government and education. However, this is not new. “What should be the relationship between the 
federal and state governments?” was the heart of debate on the U.S. Constitution, as preserved in the 
Federalist and Antifederalist speeches, articles, and letters from 1787 to 1788, prior to ratification. 4 
The debate unfolds today in appeals for “systemic” reform of government, from politicians who sit 
on both sides of the “aisle” in Congress and from a host of critical, outspoken citizens. No one knows 
where the debate will lead nor, for that matter, if it will ever have a conclusion. Indeed, in the 
American constitutional government, this ongoing debate is a fundamental political process to assure 
the republic that its elected rulers will foment neither tyranny nor anarchy. Over time, emphasis on 
this political debate waxes and wanes. Today there is a broadly-based yearning for a better American 
government. Systemic reform is the by-partisan effort to lead the way. 

There are also vocal critics who call for systemic reform of public education in America. In this 
instance, systemic reform refers to changing the systems of publicly supported education. Through 
the years, there have been many large scale efforts to change public education in America. 5 In the 
decade of the 1960s, Americans implemented new curricula in response to the criticism that Russia 
was winning the space contest with Sputnik. At best, the results of the changes were mixed. 
Publication of A Nation at Risk in 1983 propelled the “first wave” of criticism of American teachers 
for their generally low level of content knowledge, for their lack of adequate pedagogical skills, and 
for the low standards accepted for students’ learning. Results of educational assessment indicated 
that U.S. students fared poorly in math and science, especially, in comparison with their peers in 
some Asian and European countries. Later in the 1980s, a “second wave” of school reform saw 
educators and public policy officials embrace organization theory in an attempt to explain what’s 
wrong with American public schools. Subsequently there was a focus on restructuring the 



4 Consult the two volume set by Bernard Bailyn,. ed., Debate on the Constitution (New York: 
Literary Classics, 1993). 

5 Maris Vinovskis, “An Analysis of the Concept and Uses of Systemic Educational Reform,” 
American Educational Research Journal, Vol. 33, No. 1, Spring 1966, pp. 53-85. 
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organization and management of schools, which led to recommendations for, and implementation 
of, decentralization and site-based management. 

Systemic reform is the “third wave” in contemporary educational reform. President George Bush 
launched the bi-partisan initiative as Goals 2000 at the historic Charlottesville Education Summit 
of 1 989. William Bennett was his Secretary of Education. Currently, President Bill Clinton continues 
this commitment as a centerpiece of his administration, with Richard Riley as the Secretary of 
Education. The chief architects of Secretary Riley’s policy toward systemic educational reform, 
Marshall Smith and Jennifer O’ Day, emphasize the federal government’s leadership role in systemic 
educational reform. 6 Its crest bares the following terms: expanded role of states in public education; 
standards-driven and intellectually challenging curriculum and assessment; and provision of truly 
equitable opportunities for all children to learn — especially in math and science. At the present time, 
the Department of Education is soliciting states’ voluntary participation in a national achievement 
testing program. 

Critics of Secretary Riley’s policy initiative claim that systemic school reform does not give 
sufficient emphasis to basic knowledge and skills or to parents’ role in the school’s decision-making 
process and to the choice of school that their children would attend. Some of the critics claim that 
there is no scientific basis for “systemic reform;” there is only a strong belief, or rationale, that 
“changing” the system of education in accordance with these normative principles will lead to 
significant increases in students’ math and science achievements, especially among poor and 
historically disadvantaged children. Its protagonists, who were sure-footed prior to the dramatic 
change in the U.S. Congress in fall 1994, now express chagrin that “the standards movement seems 
far more fragile than the future of enhanced local control.” 7 Some supporters have issued the call for 
a balancing of the old and the new across the curriculum. 8 Phonics programs that once were 
condemned to the proverbial dust bin are now recognized for their particular purpose in helping 
children learn to read. The politics of educational reform had gone so far that President Clinton said 



6 Carl Kaestle and Marshall Smith, “The Federal Role in Elementary and Secondary 
Education, 1940-1980,” Harvard Educational Review, 52, No. 4, 1982, pp. 384-408; Jennifer O’ Day 
and Marshall Smith, "Systemic Reform and Educational Opportunity,” Designing Coherent 
Education Policy, Susan H. Fuhrman, ed. San Francisco: Jossey-Bass, 1993; and Marshall Smith and 
Jennifer O’Day, “Teaching Policy and Research on Teaching.” (Paper prepared for meeting of 
Working Party on the Condition of Teaching, OECD, Paris, France, 1988); For an interesting 
critique of systemic reform and a fresh set of policy recommendations, see William Clune, “The Best 
Path to Systemic Educational Policy: Standard/Centralized or Differentiated/Decentralized?” 
Educational Evaluation and Policy Analysis, Vol. 15, No. 3, Fall 1993, pp. 233-254. 

7 Amy Wells and Jeanie Oaks, “Potential Pitfalls of Systemic Reform: Early Lessons from 
Research on Detracking,” in Sociology of Education, special issue, 1996, pp. 135. 

8 Margaret Goertz, Robert Floden, and Jennifer O’ Day, “The Bumpy Road to Education 
Reform,” a paper submitted to the second annual NISE forum, February 1997. 
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